Probing ECG-based mental state monitoring on short time segments

Electrocardiography is used to provide features for mental state monitoring systems. There is a need for quick mental state assessment in some applications such as attentive user interfaces. We analyzed how heart rate and heart rate variability features are influenced by working memory load (WKL) and time-on-task (TOT) on very short time segments (5s) with both statistical significance and classification performance results. It is shown that classification of such mental states can be performed on very short time segments and that heart rate is more predictive of TOT level than heart rate variability. However, both features are efficient for WKL level classification. What's more, interesting interaction effects are uncovered: TOT influences WKL level classification either favorably when based on HR, or adversely when based on HRV. Implications for mental state monitoring are discussed

Eye blink characterization from frontal EEG electrodes using source separation and pattern recognition algorithms

Due to its major safety applications, including safe driving, mental fatigue estimation is a rapidly growing research topic in the engineering field. Most current mental fatigue monitoring systems analyze brain activity through electroencephalography (EEG). Yet eye blink analysis can also be added to help characterize fatigue states. It usually requires the use of additional devices, such as EOG electrodes, uncomfortable to wear, or more expensive eye trackers. However, in this article, a method is proposed to evaluate eye blink parameters using frontal EEG electrodes only. EEG signals, which are generally corrupted by ocular artifacts, are decomposed into sources by means of a source separation algorithm. Sources are then automatically classified into ocular or non-ocular sources using temporal, spatial and frequency features. The selected ocular source is back propagated in the signal space and used to localize blinks by means of an adaptive threshold, and then to characterize detected blinks. The method, validated on 11 different subjects, does not require any prior tuning when applied to a new subject, which makes it subject-independent. The vertical EOG signal was recorded during an experiment lasting 90 min in which the participants’ mental fatigue increased. The blinks extracted from this signal were compared to those extracted using frontal EEG electrodes. Very good performances were obtained with a true detection rate of 89% and a false alarm rate of 3%. The correlation between the blink parameters extracted from both recording modalities was 0.81 in average

Étude de corrélats électrophysiologiques pour la discrimination d'états de fatigue et de charge mentale : apports pour les interfaces cerveau-machine passives

Mental state estimation on the basis of cerebral activity and its resulting physiological activities has become a challenge for passive Brain-Computer Interfaces (BCI), in particular to address a need in neuroergonomics. This thesis work focuses on mental fatigue and workload estimation. Its purpose is to provide efficient and realistic processing chains. Thus, one issue was the modulation of workload markers as well as classification performance robustness depending on time-on-task (TOT). The impact of workload and TOT on attentional state markers was also assessed. For those purposes, an experimental protocol was implemented to collect the electroencephalographic (EEG), cardiac (ECG) and ocular (EOG) signals from healthy volunteers as they performed for a prolonged period of time a task that mixes working memory load and selective attention. Efficient signal processing chains that include spatial filtering and classification steps were designed in order to better estimate these mental states. The relevance of several electrophysiological markers was compared, among which spontaneous EEG activity and event-related potentials (ERPs), as well as various preprocessing steps such as spatial filtering methods for ERPs. Interaction effects between mental states were brought to light. In particular, TOT negatively impacted mental workload estimation when using power features. However, the chain based on ERPs was robust to this effect. A comparison of the type of stimuli that can be used to elicit the ERPs revealed that task-independent probes still allow very high performance, which shows their relevance for real-life implementation. Lastly, ongoing work that aims at assessing task-robust workload markers, as well as the usefulness of auditory ERPs in a single-stimulus paradigm will be presented as prospects.L'estimation de l'état mental d'un individu sur la base de son activité cérébrale et de ses activités physiologiques résultantes est devenue l'un des challenges des interfaces cerveau-machine (ICM) dites passives, dans le but notamment de répondre à un besoin en neuroergonomie. Ce travail de thèse se focalise sur l'estimation des états de fatigue et de charge mentale. Son objectif est de proposer des chaines de traitement efficaces et réalistes dans leur mise en œuvre. Ainsi, un des points à l'étude a été la modulation des indicateurs de charge ainsi que la robustesse des performances de classification en fonction du temps passé sur une tâche (TPT). L'impact de la charge et du TPT sur les marqueurs d'état attentionnel a aussi été évalué. Pour ce faire, un protocole expérimental a été mis en œuvre afin de recueillir les signaux électro-encéphalographiques (EEG), cardiaques (ECG) et oculaires (EOG) de participants volontaires sains lors de la réalisation prolongée d'une tâche combinant charge en mémoire de travail et attention sélective. Des chaînes de traitement performantes incluant une étape de filtrage spatial et une classification supervisée ont été mises en place afin de classer au mieux ces états. La pertinence de plusieurs marqueurs électrophysiologiques a été comparée, notamment l'activité EEG spontanée et les potentiels évoqués (PEs), ainsi que différentes étapes de prétraitement dont les méthodes de filtrage spatial pour PEs. Des effets d'interactions ont été mis au jour entre les différents états mentaux, dont un effet négatif du TPT sur les performances en classification de la charge mentale lorsque l'on utilise des marqueurs mesurant la puissance moyenne de l'EEG dans des bandes de fréquence d'intérêt. La chaîne basée sur les PEs est en revanche robuste à cet effet. Une comparaison du type de stimuli utilisables pour éliciter les PEs a révélé que des stimuli tâche-indépendants permettent tout de même d'obtenir des performances très élevées, ce qui montre leur pertinence pour une implémentation en situation réelle. En perspective seront présentés des travaux en cours visant à mettre en évidence des marqueurs de charge mentale robustes à la tâche, ainsi que l'utilité des potentiels évoqués auditifs en paradigme de simple stimulus

Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs

Efficient Workload Classification based on Ignored Auditory Probes: A Proof of Concept

Mental workload is a mental state that is currently one of the main research focuses in neuroergonomics. It can notably be estimated using measurements in electroencephalography (EEG), a method that allows for direct mental state assessment. Auditory probes can be used to elicit event-related potentials (ERPs) that are modulated by workload. Although, some papers do report ERP modulations due to workload using attended or ignored probes, to our knowledge there is no literature regarding effective workload classification based on ignored auditory probes. In this paper, in order to efficiently estimate workload, we advocate for the use of such ignored auditory probes in a single-stimulus paradigm and a signal processing chain that includes a spatial filtering step. The effectiveness of this approach is demonstrated on data acquired from participants that performed the Multi-Attribute Task Battery – II. They carried out this task during two 10-min blocks. Each block corresponded to a workload condition that was pseudorandomly assigned. The easy condition consisted of two monitoring tasks performed in parallel, and the difficult one consisted of those two tasks with an additional plane driving task. Infrequent auditory probes were presented during the tasks and the participants were asked to ignore them. The EEG data were denoised and the probes’ ERPs were extracted and spatially filtered using a canonical correlation analysis. Next, binary classification was performed using a Fisher LDA and a fivefold cross-validation procedure. Our method allowed for a very high estimation performance with a classification accuracy above 80% for every participant, and minimal intrusiveness thanks to the use of a single-stimulus paradigm. Therefore, this study paves the way to the efficient use of ERPs for mental state monitoring in close to real-life settings and contributes toward the development of adaptive user interfaces

EEG index for control operators’ mental fatigue monitoring using interactions between brain regions

Mental fatigue is a gradual and cumulative phenomenon induced by the time spent on a tedious but mentally demanding task, which is associated with a decrease in vigilance. It may be dangerous for operators controlling air traffic or monitoring plants. An index that estimates this state on-line from EEG signals recorded in 6 brain regions is proposed. It makes use of the Frobenius distance between the EEG spatial covariance matrices of each of the 6 regions calculated on 20s epochs to a mean covariance matrix learned during an initial reference state. The index is automatically tuned from the learning set for each subject. Its performance is analyzed on data from a group of 15 subjects who performed for 90 min an experiment that modulates mental workload. It is shown that the index based on the alpha band is well correlated with an ocular index that measures external signs of mental fatigue and can accurately assess mental fatigue over long periods of time

Efficient mental workload estimation using task-independent EEG features

Objective. Mental workload is frequently estimated by EEG-based mental state monitoring systems. Usually, these systems use spectral markers and event-related potentials ( ERPs ) . To our knowledge, no study has directly compared their performance for mental workload assessment, nor evaluated the stability in time of these markers and of the performance of the associated mental workload estimators. This study proposes a comparison of two processing chains, one based on the power in fi ve frequency bands, and one based on ERPs, both including a spatial filtering step ( respectively CSP and CCA ) , an FLDA classification and a 10-fold cross-validation. Approach. To get closer to a real life implementation, spectral markers were extracted from a short window ( i.e. towards reactive systems ) that did not include any motor activity and the analyzed ERPs were elicited by a task-independent probe that required a re fl ex-like answer ( i.e. close to the ones required by dead man ’ s vigilance devices ) . The data were acquired from 20 participants who performed a Sternberg memory task for 90 min ( i.e. 2 / 6 digits to memorize) inside which a simple detection task was inserted. The results were compared both when the testing was performed at the beginning and end of the session. Main results. Both chains performed significantly better than random; however the one based on the spectral markers had a low performance ( 60% ) and was not stable in time. Conversely, the ERP-based chain gave very high results ( 91% ) and was stable in time. Significance. This study demonstrates that an efficient and stable in time workload estimation can be achieved using task-independent spatially filtered ERPs elicited in a minimally intrusive manner

Mental fatigue and working memory load estimation: Interaction and implications for EEG-based passive BCI

Current mental state monitoring systems, a.k.a. passive brain-computer interfaces (pBCI), allow one to perform a real-time assessment of an operator's cognitive state. In EEG-based systems, typical measurements for workload level assessment are band power estimates in several frequency bands. Mental fatigue, arising from growing time-on-task (TOT), can significantly affect the distribution of these band power features. However, the impact of mental fatigue on workload (WKL) assessment has not yet been evaluated. With this paper we intend to help fill in this lack of knowledge by analyzing the influence of WKL and TOT on EEG band power features, as well as their interaction and its impact on classification performance. Twenty participants underwent an experiment that modulated both their WKL (low/high) and time spent on the task (short/long). Statistical analyses were performed on the EEG signals, behavioral and subjective data. They revealed opposite changes in alpha power distribution between WKL and TOT conditions, as well as a decrease in WKL level discriminability with increasing TOT in both number of statistical differences in band power and classification performance. Implications for pBCI systems and experimental protocol design are discussed

Towards Multi-UAV and Human Interaction Driving System Exploiting Human Mental State Estimation

This paper addresses the growing human-multi-UAV interaction issue. Current active approaches towards a reliable multi-UAV system are reviewed. This brings us to the conclusion that the multiple Unmanned Aerial Vehicles (UAVs) control paradigm is segmented into two main scopes: i) autonomous control and coordination within the group of UAVs, and ii) a human centered approach with helping agents and overt behavior monitoring. Therefore, to move further with the future of human-multi-UAV interaction problem, a new perspective is put forth. In the following sections, a brief understanding of the system is provided, followed by the current state of multi-UAV research and how taking the human pilot's physiology into account could improve the interaction. This idea is developed first by detailing what physiological computing is, including mental states of interest and their associated physiological markers. Second, the article concludes with the proposed approach for Human-multi-UAV interaction control and future plans

Estimation of Working Memory Load using EEG Connectivity Measures

Working memory load can be estimated using features extracted from the electroencephalogram (EEG). Connectivity measures, that evaluate the interaction between signals, can be used to extract such features and therefore provide information about the interconnection of brain areas and electrode sites. To our knowledge, there is no literature regarding a direct comparison of the relevance of several connectivity measures for working memory load estimation. This study intends to overcome this lack of literature by proposing a direct comparison of four connectivity measures on data extracted from a working memory load experiment performed by 20 participants. These features are extracted using pattern-based or vector-based methods, and classified using an FLDA classifier and a 10-fold cross-validation procedure. The relevance of the connectivity measures was assessed by statistically comparing the obtained classification accuracy. Additional investigations were performed regarding the best set of electrodes and the best frequency band. The main results are that covariance seems to be the best connectivity measure to estimate working memory load from EEG signals, even more so with signals filtered in the beta band. point